Assessment of Learning in a Graduate Level Pedagogy Course 


W.F. Preston Cumming 
Department of Geography 
The University of Colorado Boulder 
Boulder, Colorado, 80309 

May 2012 


Abstract 

The increasing expectations for all incoming faculty to not only produce exceptional research, but also be 
better communicators and educators immediately has put the pressure on graduate students looking to 
move into academia to improve on their ability to teach. By providing the opportunity for Masters and 
PhD level students, the Departments of Geography, Ecology and Evolutionary Biology, and 
Environmental Studies at the University of Colorado Boulder are giving interested students this much 
needed enhancement of skills through an interdisciplinary skills and methods course. A pre- and post- 
assessment analysis of student learning was given to students in the course in 2011 to assess student 
understanding and potential implementation of the methods presented to them. Results of improvement 
are further assessed with the comparison of faculty course questionnaire results with departmental, 
division, and campus wide results. 
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Introduction 

Research on how well post-secondary students learn has predominantly been conducted in 
undergraduate classrooms (Cook, Kennedy, and McGuire 2013; Bridges et al. 2002; Brookheart 2001; 
Koljatic and Kuh 2000). The lack of assessment and dissemination of how a graduate student takes in new 
information is a clear gap in understanding of how graduate students learn. Since a good proportion of 
these students continue on to academic or leadership positions it is imperative that an evaluation of the 
effectiveness of their learning is conducted. By preparing our graduate students to be better classroom 
managers and effective communicators through course work designed to instruct them on proper 
methodology and management, we are assuring their success as well as the success of the undergraduate 
and graduate students and employees they are charged with leading. These graduate students will face 
challenges significantly different from those of their predecessors, including working with students and 
employees who represent increasingly diverse ethnicities and academic backgrounds (Austin 2002; Cody 
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and Hagerman 1997). In order to succeed in future professions or academe, graduate students will need 
opportunities to engage with the scholarship of teaching in a more diverse climate (Smith, Wenderoth, 
and Tyler 2013; Trautmann and Krasny 2006; Golde and Dore 2001). This will not only lead to them 
being better educators, but has also been shown to increase research skills in graduate students (Feldon et 
al. 2011). 

The idea behind classroom assessment involves student and teachers in the continuous monitoring 
of students' learning (Jacoby et al. 2014; Bell and Cowie 2001; Black 1998; Sadler 1998). It provides 
faculty with feedback about their effectiveness as teachers, and it gives students a measure of their 
progress as learners. Most important, because classroom assessments are created, administered, and 
analyzed by teachers themselves on questions of teaching and learning that are important to them, the 
likelihood that instructors will apply the results of the assessment to their own teaching is greatly 
enhanced. An excellent approach to evaluating increases in student understanding and knowledge gain is 
a pre- and post-assessment. Pre- and post-assessment designs are widely used in educational research, 
primarily for the puipose of comparing groups and/or measuring change resulting from teaching 
technique changes (Gosselin and Macklem-Hurst 2002; Zimmerman and Williams 1998; Dugard 1995). 

In educational research, change is commonly measured in such variables as meeting of course goals and 
objectives, student understanding of processes, and student ability to obtain higher levels of thinking. The 
measurement of change provides a vehicle for assessing the impact of pedagogical strategies and 
processes, as well as the effects of specific methodologies and interventions. 

Pedagogy courses are designed to teach students how to teach within their own and related 
disciplines. In the pedagogy course described here students were asked to take pre- and post-assessments 
composed of ten different questions to evaluate participant knowledge and confidence in their abilities to 
institute pedagogical methods into their classes. The questionnaire was given the first day of the course 
and then the last day of the course in order to assess the overall learning of the students of the methods 
presented to them. The objective of this study was to gain immediate feedback on the courses ability to 
educate graduate students about pedagogical practices for effective teaching and learning in the classroom 
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as well as evaluate the increase in the particular students’ in-class effectiveness after taking the course. 
This course presents a unique case for assessment in that it analyzed student learning of various 
pedagogical concepts of which the students were then able to utilize in their own classrooms. 

Course description 

This graduate level pedagogy course was first taught in the fall of 2010. At that time no Institutional 
Review Board (IRB) approval was sought for the distribution of the questionnaire, as it was used solely 
for internal use as a tool for further development of the questionnaire. The course was first taught with a 
single instructor of record, a PhD student in the Environmental Studies program sponsored by a faculty 
mentor who oversaw course development, along with the Director of the Graduate Teacher Program 
(GTP) at the University of Colorado (www.colorado.edu/gtp); and grading. The course was listed solely 
within the Environmental Studies department initially, however it was open to and advertised among 
many departments on campus. The 2011 version of this course was co-taught by three experienced PhD 
students who were the GTP Leads, who led individual efforts for the GTP within their individual 
departments, from Environmental Studies, Evolutionary Ecology and Biology, and Geography and was 
cross-listed between the three in the fall of 201 1 and as of fall of 2014 was still being taught in these 
departments. The interdisciplinary focus of the pedagogy course gave new insight into the class as well as 
altered the course to achieve the goals and objectives they believe to be important in a pedagogy course at 
the graduate level. Institutional Review Board approval was obtained for this study in the summer of 
2011 . 

Student demographics 

Study participants consisted of 18 students (6 male and 12 female) that met once a week for a 1 -credit 
course. Students ranged in classroom experience (1-7 semesters in the classroom). The mix of students 
enrolled also varied in terms of their current graduate program standing, with 9 being in an M.S. program 
and 9 a PhD program. Many students expressed interest in teaching after their graduate careers with even 
15 students being interested in teaching at a liberal arts school or a research institution. It is interesting to 
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note that the few remaining students (3) indicated that teaching was not in their career plans, but the 
knowledge that they will gain could only benefit them in the job place. 

Research design and methods 

The course was designed to provide students with the knowledge and skills required for implementing 
new techniques for improving their teaching skills and designing effective class materials. Teaching 
strategies for the course included in-class discussions, online discussions, course unit preparation, and 
lectures. The class requirements consisted of participation in discussions both in class and in an online 
WIKI format and a group project in which students produced a course unit for a particular course 
surrounding a topic of their choosing then presented the unit to the class. 

A pre and post-assessment to quantitatively and qualitatively analyze the gains in understanding 
and implementation within each type of pedagogical method as well as comparisons between types of 
methods was utilized. Evaluation through pre- and post-assessment assessment is used in many 
disciplinary contexts (Crisostomo 2010; Enos 2010; Robelia et al. 2010; Beers 2005), this format is 
normally selected to measure student -learning outcomes in each course type. The pre- and post- 
assessment format allows instructors to determine which learning outcomes they wish to test for and 
develop a testing instrument accordingly (University of Wisconsin-Madison 2009; Asians and Rossi 
2005). This type of assessment assumes that improvement occurs between the pre- and post-assessment 
and that this improvement can be attributed to learning that takes place over the time period of instruction, 
whether from in-class instruction or out of class learning (Askins and Rossi 2005). These studies 
generally utilize a single pedagogical method that may not be the norm within that discipline to assess 
whether student learning increases through various methodologies. 

The current study fills a gap in knowledge by utilizing a varying approach to these same methods 
in order to assess graduate student learning of these diverse pedagogical methods in order to apply them 
to their own current or future courses of instruction. For this, we used a simultaneous mixed method 
design, where both the quantitative and qualitative data were collected and analyzed at the same time 
(Teddie and Teddie 1998). In addition to this, a paired t-test was performed on the coded data in order to 
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compare the repeated measure (i.e. the pre- and post-assessment) on the same group in order to determine 
whether the means differ. The responses to open-ended questions are the most raw and unaffected parts of 
this and any survey analysis. The questions are completely unaided, and respondents can say or write 
anything that comes to mind. They are not limited to the selecting choices or guided in their response. 
With open-ended questions you get a true sense for how the respondents feel and how to most effectively 
adjust course material and design for future classes. 

Finally, to assess the improvements made by students in this course, Faculty Course 
Questionnaire data freely obtained on the University of Colorado Boulder’s Planning, Budget, and 
Analysis program (available at www.colorado.edu/pba/fcq) was used. Departmental, Division, and 
Campus wide averages for several categories (course and instructor overall, instructor effectiveness, and 
how much students learned) were used as proxies for student learning and provide an effective 
comparison with courses taught by graduate students that have not taken this course. As no department 
that this course is currently open to provides a similar course, the comparisons should provide a useful 
comparison. 

Results and discussion 
Likert Scale Results 

The questionnaire (see Appendix 1) was designed in order to determine the level of student learning in the 
course. Each student was asked to rank their ability to implement a pedagogical method in their classroom 
and then was given the opportunity to expand on that answer in an open-ended format. The Likert Scale 
was utilized for the initial responses with 1 being disagree, 2 being somewhat disagree, 3 being neither 
agree nor disagree, 4 being somewhat agree, and 5 indicating agreement. The pre- and post-assessment 
questions are described in Table 1 . For every question there was a significant increase in knowledge by 
each student. The only exception was question 1, in which the increase of 0.44 (p value of 0.14) was not 
significantly different. This is most likely due to the highest pre -evaluation value of 3.89 compared to all 
other questions. 

Open-ended question results for pre- and post-assessment 


5 



In order to assess student learning further in the course, open-ended questions were associated with many 
of the pre- and post-assessment questions (see Appendix 1). 

Question 1A: Could you please name and describe any learning styles that you are aware of? 

Answers to this question on the pre-evaluation were varied across learning styles. The most widely 
discussed learning styles of visual, auditory, and kinesthetic had the most responses with 10, 11, and 10 
respectively, with active (4), passive (3), concrete (1), abstract (3), and logical (1) having very few 
occurrences in the discussions. The lecture on learning styles was coordinated with a discussion of 
various styles, but with an emphasis on the Kolb learning style inventory in particular. Kolb saw learning 
style as the unique learning method presented by the learner during the learning process and situation 
(Wang et al. 2006). It was interesting to note that the Kolb styles were discussed four times in the pre- 
evaluation. This was most likely due to the Graduate Teacher Program’s influence and attendance 
throughout the orientation week leading up to classes during the fall semester as well as the range of 
experience of the graduate students in the course. 

The post-assessment returned with a similar outcome, but with fewer categories of responses. 
Visual, auditory, and kinesthetic learning all were discussed on 12 of the responses, while Kolb, 
interestingly was only discussed on three. The number of responses that did not recall was also three. 
Having the Kolb learning styles discussed in fewer responses may indicate that there needs to be more 
direct reference and maybe even a requirement of all students to take the inventory to have it better 
outlined for them. Although the directive of the class period may not have been met, the students did 
seem to respond well to various learning style discussions. 

Question 3 A: Could you please name and describe any activities that would engage students in 
classroom discussions? 

Answers to this question varied widely on both the pre- and post-assessments. Most of the responses on 
the pre-evaluation were based on group discussions (10 responses) and question and answer times during 
the class period (6 responses). However, there were numerous other responses including using lecture, 
videos, debates, games, and think-pair-share activities that all were discussed at least once within the 
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answers. This class period focused on the think-pair-share and gallery walk activities with a specific 


discussion thread opened up on the WIKI to include any other ideas for classroom discussion. 

Within the post-assessment answers it is evident that the class objective was met with a 
significant increase in responses for think-pair-share (1 1), gallery walk (13), and discussion based 
activities (13) respectively. The diversity in answers significantly increased to now include question and 
answer time (2), debates (3), role playing (5), games (2), case studies (3), classroom research projects (1), 
presentations (1), and videos (1). This may be due to the WIKI posts for activities as well as the 
classroom discussion following the two activity demonstrations in class. 

Question 4A: Could you please write a question for each of Bloom’s hierarchy? 

One of the most significant increases in correct responses from the pre to the post-assessment occurred on 
the Bloom’s Taxonomy question. Answers to the pre-evaluation were very low with knowledge (2), 
comprehension (2), application (3), analysis (3), synthesis (2), evaluation (1), and unsure (5) showing 
little or no understanding of the taxonomy or at least a lack of recall from attended Graduate Teacher 
Program workshops. The post-assessment significantly increased with knowledge (16), comprehension 
(9), application (9), analysis (12), synthesis (9), evaluation (7), and unsure (5). It is also important to note 
here that only one student was able to name all levels specifically. 

Student responses to this question were varied. However, every student with their notes in front 
of them was able to accurately identify and utilize all of the taxonomic levels within their course units for 
the final project. Many of the students who replied partially or not at all were explicit in identifying this 
fact. It is also interesting to notice that most students could accurately identify and write a question for the 
basic level (knowledge), but the numbers of correct responses decreased as the definition and questions 
became more difficult to describe. 

The class period devoted to this assessment technique was designed not only to discuss Bloom’s 
taxonomy, but also other modes of assessment. Students were given the opportunity to write questions 
and have them “graded” for correctness. Only a handful of students took this opportunity and it may be 
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part of the reason for not completely grasping the concepts at every taxonomic level. However, the 
classroom discussion and WIKI thread were both fruitful and highly useful to all students and instructors. 

Question 5 A: Describe how you would implement a classroom activity that uses the Problem 
Orientation as a framework for helping students analyze a problem. 

The discussion on the Problem Orientation framework was designed to have students look at how 
instructors can approach an interdisciplinary course that is focused, whether one class period or 
throughout a unit, on solving a larger problem such as climate change or global health care using a four 
step method that includes the goals of the discussion, current trends in the issue, conditions of both sides 
of the issue, projections of outcomes, alternatives to the outcomes. Many students were Environmental 
Studies graduate students and may have been able to answer this question prior to the course. However, 
there were no responses on the pre-evaluation indicating either a misunderstanding of the question or it 
was not posed in a way to have them comprehend the meaning. 

After the course, there was a significant increase in student’s comprehension of the question and 
their ability to describe how they would approach the issue. Many students discussed the use of case 
studies (4), describing and utilizing both sides of an issue (4), using real world problems (4), but only a 
few indicated all of the steps involved in the framework (2). Only one student indicated that they did not 
remember what the framework was or begin to describe how they would approach a problem oriented 
class. 

Question 6A: Describe what a learning community is and how you would encourage students to 
solve a current problem using one. 

The pre-evaluation only had one student who accurately described what a learning community was. 
Whether this was based on prior knowledge or a good guess is undetermined. The Center for the 
Integration of Research, Teaching, and Learning (CIRTL) describes Teaching-as-Research (TAR) as 
“involving the deliberate, systematic, and reflective use of research methods to develop and implement 
teaching practices that advance the learning experiences and outcomes of students and teachers (CIRTL 
2012). The treatment of a classroom as a laboratory for determining outcomes based on the scientific 
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method is a natural setting for many STEM students. The determination of what students have actually 
learned in the classroom is at the core of teaching and. therefore, at the core of TAR. Instructors have an 
exceptional opportunity to use their classrooms as laboratories for the study of learning and through such 
study to develop a better understanding of the learning process and the impact of their teaching upon it. 

By beginning with constructing a foundation of knowledge and progressing through the framework of 
TAR as follows: 

• Learning foundational knowledge. 

• Creating objectives for student learning. 

• Developing a hypothesis for practices to achieve the learning objectives. 

• Defining measures of success. 

• Developing and implementing teaching practices within an experimental design. 

• Collecting and analyzing data. 

• Reflecting, evaluating, and iterating (CIRTL 2012). 

The current study follows these steps and begins to assess the learning ability of graduate students 
in a pedagogy course. The novelty of this research is defined by the ability of each of these students to 
potentially utilize the techniques in their own classrooms and potentially have TAR projects conducted in 
their own classrooms using each individual pedagogical technique they have now discovered. After the 
discussion of this basic CIRTL network pillar in the class period, 13 of the 18 students were able to define 
what a learning community was and 12 students were able to describe how they would use it to encourage 
students to solve a current problem. Many students either defined learning community or explained how 
they would approach it, with very few doing both. 

Question 7A: Could you describe how you would integrate diverse issues and viewpoints into a 
problem-based curriculum by giving one example of a diverse issue? 

The pre-assessment answers to this question varied with only five total responses including energy use 
(1), pine beetle infestation (1), open space issues (1), water rights (1), and politics (1). The post- 
assessment showed a significant increase in the diversity and response rate including climate change (5), 
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health care (3), evolution (2), ranching (1), mining (1), developing country’s rights (1), and population 
control (1). 

Question 8A: Could you describe what TAR is and how it is used as an assessment tool? 

The pre-assessment results for this question had no responses. This indicated that students were not 
exposed to this terminology or topic. Utilizing this discussion as a second class period for assessment 
showed in the post-assessment with increases in the definition of TAR (7), the use of this pre and post- 
assessment as a method for a TAR (8), the discussion of scientific research (3), and using TAR as an 
assessment tool (7) all being discussed. Although the specifics of TAR were not given as much time 
during the class period as would have been preferred, students did seem to grasp the concept and were 
able to see how the scientific method can be used as a means for assessment in a classroom. 

Question 9A: Could you describe what the difference between classroom and course assessment is? 
The answers on the pre-evaluation showed that only one student could accurately define what classroom 
assessment is and only one (different) student could define course assessment. Through the classroom 
discussion, the WIKI posts, and the development of their individual projects, those numbers significantly 
increased to 1 1 students in each category (many of which were the same) in the post-assessment. 
Question 10A: Could you outline (bullet points) how you would go about designing a problem- 
based course that uses and interdisciplinary framework to highlight themes that are common 
among problem categories? 

Within the pre -evaluation only one student attempted to outline how they might go about designing an 
interdisciplinary course. This number increased to 15 students being able to outline a course accurately 
with only three students failing to answer the question for the post-assessment. 

The increase in student responses to the open-ended questions indicates the ability of graduate 
students to learn how to become more effective teachers in their classrooms. Many of these students were 
either first or second time instructors and were potentially very nervous about either being in front of 
students for the first time or wanting to provide the best experience they could for the undergraduate 
students. This may indicate why these students enrolled in this type of course. The answers and 
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explanations provided in this study will help direct the future of this pedagogy course in general and will 
provide information necessary to help other pedagogy courses based on this concept in other departments. 

Finally, by comparing the graduate student’s faculty course questionnaire results from courses 
taught prior to taking the course and afterwards we see how effective such a course can be. Previous to 
taking the course all students rated lower than the department, division, and campus overall in course 
overall, instructor overall, effectiveness, and only slightly higher than the department specifically in how 
much students learned in the course (Table 2). However, after taking the course, the average student 
results for all four categories show significant improvement over prior to taking the course and become 
higher than department, division, and campus averages (Table 2). Several factors may be involved in this 
increase including increased experience in the classroom and comfort with the material if in similar 
courses. However, it may also be attributed to the increased resources provided through a course designed 
to improve the graduate student’s ability to teach. 

This project represented an attempt to move beyond the usual research on undergraduate 
education into independent assessment of student learning that can be translated directly into performance 
in the classroom. The findings show that students were able to take what they were learning about 
instruction and apply it to real world situations in their own classrooms. The overall instructional 
methodology grounded in the utilization of the same techniques the instructors were trying to teach the 
students, was responsible for engaging the students in active learning, creation of learning communities, 
and the increased awareness of learning through diversity. 

Conclusions 

The results of this study show that there may be many improvements made to the evaluation and in turn 
the course structure as a whole the further improvement of this and all pedagogy courses needs to be a 
constant process. The University as well as the educational community in general will benefit from this 
research, as the results will supplement literature on the assessment of teaching and learning in graduate 
level courses. Results from this research are being used currently for the development of similar courses 
in disciplines where graduate students are intending on either becoming faculty at an institute of higher 
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education or in leadership roles in industry, government, or non-governmental organizations where 
having a Master’s or PhD level education injects you directly into a role as an instructor. 
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Table 1. Likert Scale answers for pre- and post-assessment data 

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 


n 

18 

18 

18 

18 

Pre-mean 

3.89 

2.89 

2.83 

2.25 

Post-mean 

4.33 

4.78 

4.56 

4.61 

Difference 

0.44 

1.89 

1.72 

2.36 

Increase 

9% 

38% 

35% 

47% 

p-value 

0.14 

<.001 

<.001 

<.001 


18 

18 

18 

18 

18 

1.89 

2.28 

2.72 

1.41 

2.06 

3.89 

3.78 

4.39 

3.33 

3.75 

2.00 

1.50 

1.67 

1.92 

1.69 

40% 

30% 

33% 

38% 

34% 

<.001 

<.001 

<.001 

<.001 

<.001 


Table 2. Faculty Course Questionnaire responses 



Course 

Instructor 

Instructor Effectiveness 

How Much Learned 

n 

18 

18 

18 

18 

Pre 

4.23 

4.89 

4.50 

4.36 

Post 

4.57 

5.22 

4.85 

4.59 

Difference 

0.34 

0.33 

0.35 

0.23 

Increase 

5.6% 

5.5% 

5.8% 

3.8% 

Department 

4.40 

5.10 

4.65 

4.35 

Division 

4.43 

4.98 

4.53 

4.48 

Campus 

4.48 

5.00 

4.58 

4.50 


18 

2.00 

4.25 

2.25 
45% 
<.001 
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Appendix 1 

Questionnaire 

Part 1 . Please complete the following survey by circling your answer according to this scale: 

1 - Disagree 

2 - Somewhat disagree 

3 - Neither agree nor disagree 

4 - Somewhat agree 

5 - Agree 

1. 1 can describe several learning styles that are common among students. 

1 2 3 4 5 

la. If you agree with number one, could you please name and describe any learning styles that you are 
aware of. 

2. 1 can design a lesson plan that encourages each of these learning styles. 

12 3 4 

3. 1 know how to lead several different activities that engage all students in a classroom discussion. 

1 2 3 4 5 

3a. If you agree with number three, could you please name and describe any activities that would engage 
students in classroom discussions. 

4. 1 can use Bloom’s Taxonomy as a guide for writing quiz/test questions that target different levels of 
students’ understanding of a concept. 

1 2 3 4 5 

4a. If you agree with number four, could you please write a question for each of Bloom’s hierarchy. 

5. 1 can implement a classroom activity that uses the Problem Orientation as a framework for helping 
students analyze a problem. 

1 2 3 4 5 

5a. If you agreed with number five, describe how you would achieve this? 

6. 1 can design a classroom activity that encourages students to work in learning communities to devise a 
solution to a current problem. 

1 2 3 4 5 

6a. If you agreed with number six, describe what a learning community is and how you would achieve 
this? 
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7. 1 know how to integrate diverse issues and viewpoints into a problem-based curriculum. 

1 2 3 4 5 

7a. If you agree with number seven, could you describe how you would achieve this goal by giving one 
example of a diverse issue. 

8. 1 know how to practice Teaching as Research (TAR) as a method of assessing student learning. 

1 2 3 4 5 

8a. If you agree with number eight, could you describe what TAR is and how it is used as an assessment 
tool. 

9. 1 understand the differences between classroom assessment and course assessment. 

1 2 3 4 5 

9a. If you agree with number nine, could you describe what the difference between classroom and course 
assessment is. 

10. 1 feel that I can design a problem-based course that uses and interdisciplinary framework to highlight 
themes that are common among problem categories. 

1 2 3 4 5 

10a. If you agree with number ten, could you outline (bullet points) how you would go about designing 
this problem-based course. 
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